Pseudo Orthogonal Bases Give the Optimal Solution to Active Learning in Neural Networks

Pseudo orthogonal bases are a certain type of frames proposed in the engineering field, whose concept is equivalent to a normalized tight frame in the frame terminology. This paper shows that pseudo orthogonal bases play an essential role in neural network learning. One of the most important issues in neural network learning is “what training data provides the optimal generalization capability?”, which is referred to as active learning in the neural network community. We derive a necessary and sufficient condition of training data to provide the optimal generalization capability in the trigonometric polynomial space, where the concept of pseudo orthogonal bases is essential. By utilizing useful properties of pseudo orthogonal bases, we clarify the mechanism of achieving the optimal generalization.